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SYSTEM AND METHOD FOR ON-LINE ESSAY EVALUATION 

CROSS REFERENCE TO RELATED APPLICATIONS 

This application is related to U.S. Provisional Patent Application Serial 
No. 60/039,383, filed March 21, 1997, entitled "On-Line Essay Evaluation System," 
and U.S. Provisional Patent Application Serial No. 60/071,893, filed January 20, 
1998, entitled "On-Line Essay Evaluation System," the contents of which are hereby 
incorporated by reference in their entirety. 

FIELD OF THE INVENTION 

The present invention relates to systems for on-line evaluation of essay 
responses to examination questions. More specifically, the invention relates to a 
system for the submission of essays to experienced readers/graders for evaluation and 
for providing consistent, instructional feedback to students. 

BACKGROUND OF THE INVENTION 

The use of standardized testing has increased significantly in recent 
years. Indeed, standardized tests are now an essential part of almost every admission 
or qualification procedure. A recent trend in standardized testing emphasizes a move 
beyond traditional multiple choice tests in favor of tests that require open-ended 
responses such as essay responses. These open-ended responses are often referred to as 
constructed responses (CRs). CRs are not limited to written text, but may include 
graphics, videotaped performances, audio responses, as well as other forms of 
responses. In order to improve the efficiency of scoring large scale standardized tests, 
both those offered at periodic administrations as well as those offered essentially on a 
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daily basis, computer systems have been developed to automatically score multiple- 
choice responses and other simple response types. While some automatic scoring 
systems have been designed to score particular types of CRs (see e.g., U.S. patent 
application serial no. 08/794,498, entitled Automatic Scoring for Categorized Figural 
5 Responses, assigned to the same assignee hereunder), the evaluation of CRs is 

particularly well-suited to human raters. For this reason, certain computer scoring 
systems have been developed to facilitate and automate the electronic transmission of 
CRs to human raters for evaluation and scoring. However, these conventional CR 
transmission systems currently have many disadvantages. 

10 Generally, essay evaluation systems operate in the context of a particular 

test administration. The systems are used to collect, distribute, and grade actual test 
responses for a particular test. There is not presently available a system which 
provides the capability of practicing test taking skills, demonstrating content mastery, 
and receiving constructive feedback from qualified faculty who have scored actual test 

15 responses. Such a system would have tremendous benefits for those persons interested 
in improving essay writing scores and those preparing for tests. Thus, there is a need 
for a system that allows persons to improve testing skills by formulating responses to 
actual test questions in a practice setting and having their responses evaluated by actual 
test evaluators. 

2 0 Current evaluation systems also lack an adequate means for scheduling 

readers/ graders. For a system which can accept large numbers of constructed 
responses, a large number of readers are required to evaluate the constructed 
responses. Such a system should track when readers are scheduled to be on the system 
and if scheduled readers do not log on or grade a sufficient number of essays, to notify 

2 5 backup readers. 

Some prior systems permanently associate multiple essays or constructed 
responses with a particular evaiuator or reader. The constructed responses remain 
associated with the reader regardless of whether the reader is logged onto the system. 
In such systems, when the reader is not logged-in, the constructed responses sit idle 

3 0 waiting for the reader to evaluate the response. A constructed response may wait in a 

reader's queue for several days before the reader evaluates the response. Under such a 
system it is not possible to control when a constructed response may be evaluated. 
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Thus, there is a need in the art for a system for evaluating constructed responses that 
provides automated work load leveling of queued essay responses so that essay 
responses are evaluated and returned to the submitter or student within a predetermined 
^time. 

5 m A further shortcoming of prior systems is the inability to prioritize 

' constructed responses for grading. There is not presently available a means to expedite 
grading of particular constructed responses. Further, current systems do not account 
for such prioritization markings in the routing of constructed response to readers so 
that the desired deadlines are satisfied. Thus, there is a need in the art for a system 
10 which allows users to prioritize their essays and which automatically routes essays to 
evaluators so that the desired dates for evaluation are met. 
k Another shortcoming in the art is the inability to automatically determine 

dreader compensation based on work performed. Typically, in prior systems, a reader 
^or evaluator is paid a flat fee for participation in a particular scoring project. In an 
15 ^optimal system, a reader's compensation is based upon the reader's work product, i.e., 
the number of constructed responses that were graded. Further, in an optimal system, 
the compensation might vary between questions. Thus, there is a need for a system 
which provides a means for calculating compensation based on the work actually 
performed by the reader. 
2 0 An additional shortcoming in present systems is the inability to monitor 

reader evaluations so as to ensure consistency and quality in scoring. Without such a 
mechanism, scoring is random and greatly diminishes the usefulness of the system to a 
student who needs consistent scoring in order to measure improvement. Thus, there is 
a need for a system whereby reader evaluations are monitored so as to insure quality 

2 5 and consistency in grading. 

Still a further shortcoming in present systems is the inadequate feedback 
•that is provided to a student or user who has submitted an essay. Specifically, present 
systems lack the capability to provide consistent essay feedback from multiple readers 
and fail to provide students with samples of previously graded essay responses. 

3 0 Typically, in present grading systems, a single numerical score is assigned to a 

constructed response. In such systems, the user receives no feedback other than an 
overall score. Further, in those systems that provide additional feedback other than an 
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overall score, the responses are not consistent and non-standardized between readers; 
generally, the feedback comprises free-hand comments by the reader. The lack of 
consistent feedback diminishes significantly the benefit of the scoring system to the 
user or test-taker. Additionally, present systems do not provide the user with samples 
5 of scored essays for the same question. Thus, there is a need in the art for a scoring 
system that provides consistent feedback and sample essay responses. 

SUMMARY OF THE INVENTION 

Methods and systems for on-line essay evaluation in accordance with the 
present invention addresses the above described and other shortcomings in the art. 

10 According to one aspect of the invention there is provided a method for on-line 
evaluation of a constructed response to an examination question. The method 
comprises the following steps: accessing a web site via the Internet, ordering an 
evaluation account, and selecting an examination question; an examinee constructing an 
essay response to the examination question; electronically submitting the essay response 

15 to the web site for evaluation; positioning the submitted essay response in a queue of 
essay responses that need to be evaluated by a grader, the essay being placed in the 
queue based upon at least one of the time of submission of the essay response and the 
date an evaluation of the essay response is due to the examinee; the grader evaluating 
the essay response in accordance with calibrated grading guidelines, selecting an 

2 0 overall evaluation, one or more pre-defined feedback comments, and a score; and 
releasing for retrieval by the examinee the overall evaluation and the pre-defined 
feedback comments regarding the essay response; 

According to another aspect of the invention there is provided an on-line 
essay evaluation system for submitting essay responses for evaluation by graders and 

2 5 retrieving an evaluation of the essay response from the grader for submission to the 

examinee. The system comprises the following items: a database which stores a 
plurality of questions which elicit essay responses from an examinee; an essay response 
submission system which selects a question from the database and which submits an 
essay response to the selected question; an essay response administrative system which 

3 0 stores essay responses from a plurality of examinees in a queue and selectively 

distributes the queued essay responses to graders for evaluation; an automated grading 
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system which enables a grader to evaluate an essay response, to select an overall 
evaluation, one or more pre-defined additional comments, and a score; and an 
evaluation delivery system which releases to the examinee for the examinee's retrieval 
the overall evaluation and any pre-defined additional comments from the grader. 

5 The essay response administrative system schedules graders to evaluate 

queued essay responses so that essay responses are evaluated and feedback is stored 
and released for retrieval by the examinee within a predetermined time period. The 
essay response administrative system includes software which performs the following 
operations: identifies the number of essay responses which need to be evaluated in an 

0 upcoming time period; identifies the number of essay responses which are scheduled to 
be evaluated in the upcoming time period; if the number of essay responses- that need 
: td^be evaluated in the upcoming time period is less than the number of essay responses 
that are scheduled to be evaluated in the upcoming time period, identifies the number 
o£ essay responses which need to be evaluated in the upcoming time period; and if the 

5 number of essay responses that need to be evaluated in the upcoming time period is 
greater than the number of essays that are scheduled to be evaluated in the upcoming 
time period, electronically notifies backup graders of the need to provide assistance in 
the upcoming time period. 

The essay response administrative system prioritizes the presentation and 

0 distribution of essay responses to graders so that essay responses are evaluated and 
feedback is stored and released for retrieval by examinees within a predetermined time 
period. The essay response administrative system includes software which performs 
the following operations: identifies the date and time an essay response was submitted; 
identifies whether the essay response is to be graded on a regular schedule or a rush 

5 schedule; calculates a date for which an evaluation of the essay response is due based 
upon the date the essay response was submitted and whether the essay response is to be 
evaluated on a regular schedule or a rush schedule; and places the essay response in a 
queue of essay responses that is ordered by the date on which an evaluation of each 
response is due. 

0 The essay response administrative system automatically calculates any 

compensation for the graders based upon the number of constructed responses the 
grader has evaluated. The essay response administrative system includes software 
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which performs the following operations: identifies a grader; identifies the total number 
of essay responses read by the grader; and calculates the compensation for the grader 
as function of the number of essay responses read. 

The essay response administrative system includes software which 
5 performs the following operations: identifies that a grader has submitted an overall 
evaluation and score for an essay response; forwards the essay response and overall 
evaluation to a grader leader for review; and if the overall evaluation and score are not 
consistent with scoring and evaluation standards, allows grader leader to assign a new 
-score and~evaluation™so long as the supervisory grader performs- the review hra - 
10 predetermined time period. 

The automated grading system presents an essay response to a grader so 
that the grader may evaluate the essay response; and presents evaluation categories to 
the grader so that the grader may select an overall evaluation. The automated grading 
system enables a grader leader to view the overall evaluation and pre-defined additional 
1 5 comments of the grader and assign a new overall evaluation and pre-defined additional 
comments if desired. 

The evaluation delivery system releases to the examinee for the 
examinee's retrieval at least one exemplary essay response, the overall evaluation, and 
the pre-defined additional comments. The evaluation delivery system groups the 

2 0 examination question, scoring guidelines, sample essay responses, the overall 

evaluation, and pre-defined additional comments into a feedback package; stores the 
feedback package; and releases the feedback package for retrieval by the examinee. 

According to another aspect of the invention there is provided an 
Internet based system for submitting and evaluating essay responses to examination 
25 questions. The system comprises the following elements: a server computer which 
forwards examination questions to examinees and accepts essay responses to the 
examination questions for evaluation, the server being connected to the Internet; a first 
computer connected to the server computer so as to permit examinees to set up an 
account for purchasing access to an examination question and evaluation of an essay 

3 0 response submitted in response to the examination question; and a second computer 

connected to the server computer so as to permit graders to read an essay response that 
was submitted by an examinee, evaluate the essay response by providing an overall 
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comment and pre-defined additional comments, store the overall comment and the pre- 
defined additional comments, and release the overall comment and the pre-defined 
additional comments for retrieval by the examinee via the server computer. 

According to another aspect of the invention there is provided a method 
5 for on-line evaluation of constructed responses to an examination question over a 
system having a server computer connected to the Internet, a first computer 
electronically connected to the server computer, and second computer connected to the 
server computer. The method comprising the following steps: an examinee accessing 
the server computer from the first computer to set up an account for purchasing access 

10 to an examination question and evaluation of an essay response which is submitted in 
response to the examination question; an examinee accessing the server computer from 
the first computer to submit an essay response in response to the examination question; 
accepting the essay responses at the server computer; and a grader accessing the server 
computer from the second computer to read the essay response submitted by the 

15 examinee, evaluate the essay response by providing an overall comment and pre- 
defined additional comments, storing the overall comment and pre-defined additional 
comments, and releasing the overall comment and pre-defined additional comments for 
retrieval by the examinee. 



BRIEF DESCRIPTION OF THE DRAWINGS 
2 0 A full understanding of the invention can be gained from the following 

description of preferred embodiments when read in conjunction with the accompanying 
drawings in which: 

FIGURE 1 provides an operational diagram for the essay evaluation 
system of the invention; 

2 5 FIGURE 2 depicts the architecture of the system of FIGURE 1; 

FIGURE 3 provides a diagram of the functional components of the 
system of FIGURE 1; 

FIGURE 4 provides an entity relationship diagram for the system data 

dictionary; 

3 0 FIGURE 5 represents several of the functional operations provided by 
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the system of FIGURE 1; 

FIGURE 6 provides a flowchart of the essay submission and evaluation 
process provided by the system of FIGURE 1 ; 

FIGURE 7 illustrates an introductory system screen through which a 
5 student or user may access the system's user functionality; 

FIGURE 8 illustrates a system screen providing directions for using the 
system of FIGURE 1; 

FIGURE 9 illustrates a system screen which provides fee information for 
using the system; 

0 FIGURE 10 illustrates a system screen which provides sample feedback 

information similar to that which would be provided upon submission and evaluation of 
an essay; 

FIGURE 1 1 illustrates a system screen which provides the topics for 
which a user can prepare and submit essays; 
5 FIGURE 12 illustrates a system screen which allows a user to order an 

access code to be used in submitting essays; 

FIGURE 13 illustrates a system screen which provides information on 
writing an essay for evaluation; 

FIGURE 14 illustrates a system screen which allows a user to submit an 
0 essay for evaluation; 

FIGURE 15 illustrates a system screen which allows a user to access 
feedback regarding an essay; 

FIGURE 16 illustrates a system screen which allows a user to access 
responses to frequently asked questions; 
5 FIGURE 17 illustrates a system screen which allows a user to access 

information on the operation of the system; 

FIGURE 18 provides a flow chart of the essay monitoring and reader 
scheduling process; 

FIGURE 19 provides a flow chart of the work load leveling process of 
0 the system of FIGURE 1 ; 

FIGURE 20 provides a flow chart of the process for calculating reader 
compensation in the system of FIGURE 1 ; 
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FIGURE 21 provides a flow chart of the feedback process of the system 

of FIGURE 1; 

FIGURE 22 illustrates a system screen which a reader may encounter 
s\ when logging into the system for purposes of scoring essays; 

5 : FIGURE 23 illustrates a system screen that may be presented to a reader 

?- who selects to view a system training pack; 

FIGURE 24 illustrates a system screen that may be presented to a reader 
who selects to view the question to which the user's essay is a response; 

FIGURE 25 illustrates a system screen that may be presented to a reader 
10- who selects to view the scoring guidelines; 

FIGURE 26 illustrates a system screen that may be presented to a reader 
who has selected to review an essay response that has a particular score associated with 
- it; 

> FIGURE 27 illustrates a system screen that may be presented to a reader 

1B&-. who has selected to evaluate an essay; and 

FIGURE 28 illustrates a system screen that may be presented to a reader 
who has selected to review/assign pre-defined rubrics to an essay response. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

An on-line essay evaluation system with the above-mentioned beneficial 
2 0 features in accordance with a presently preferred exemplary embodiment of the 

invention will be described below with reference to FIGURES 1 through 28, It will be 
appreciated by those of ordinary skill in the art that the description given herein with 
respect to those figures is for exemplary purposes only and is not intended in any way 
to limit the scope of the invention. All questions regarding the scope of the invention 

2 5 may be resolved by referring to the appended claims. 

The present system offers students the opportunity to write practice 
essays, submit the essays to trained, expert readers, and retrieve an evaluation at the 
student's convenience. The system provides the capability for a user or test taker to 
submit essays at any time during the year, independent of the timing of an actual 

3 0 testing event. Further, the system provides the capability to prioritize essays and 

schedule readers so that essays can be evaluated on a rush basis. The essays are 
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evaluated in a manner that provides useful instructional feedback to students about their 
skills relative to the assessment or test that the student wishes to take. 

The present system provides an excellent learning opportunity and is 
unique in that the essay evaluations may be used for instructional purposes rather than 
5 strictly for evaluative purposes. Since the essay scoring and feedback is provided by 
skilled readers, the evaluation comes from an unbiased source and can prove to be 
extremely beneficial in helping students develop writing skills and demonstrate content 
mastery. Further, the system allows for submission of essay responses continuously 
throughout the year so that a student can improve and prepare for a given test 

10 administration. The student receives detailed feedback information which develops the 
student's knowledge of the subject matter and provides the student with instructional 
guidance on how to improve his or her writing skills. The student can submit essays 
in response to the same question several times and obtain feedback each time an essay 
is submitted. In this way, a user is able to measure improvement. Students can select 

15 among the various topics to explore writing techniques applicable to each topic as well 
as develop their knowledge content in particular topic areas. 

Essays are selected for presentation to a reader based on the essay topic 
and are presented in order by date and time received so that the first essays received 
are read first. Readers continue to process essays one after the other in the order the 

2 0 system presents the essays to them; readers do not select specific essays to read. 

Readers can opt to change to other topics for which they are qualified to read, thereby 
releasing any essays queued for them back into the pool to be presented to other 
readers. 

Readers schedule themselves to perform evaluations on particular 

2 5 reading days. In the preferred embodiment, Monday, Tuesday, Wednesday, Thursday, 

and Friday are separate reading days for which a reader may schedule himself or 
herself. The weekend, Saturday and Sunday, is categorized as a single reading day for 
which a reader may schedule him or herself. 

Typically, a primary reader schedules him or herself to evaluate essay 

3 0 responses to a specific question on a specific reading day. The primary reader assumes 

responsibility for evaluating a certain number of essays, if the essays are available, on 
selected days. A reader may also schedule him or herself as a backup reader whereby 
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he or she will be available on a particular reading day to read essays if the primary 
readers are unable to grade all of the essays that are scheduled for reading. Further, 
•selected readers are assigned to be scoring leaders; these highly expert leaders can 
^monitor other readers or themselves serve as a primary reader. A scoring leader 
5 * reader monitors the scores assigned by primary and backup readers for a specific 
^question on a specific reading day. 

An operational diagram for a preferred embodiment of the inventive 
system is presented in FIGURE 1 . As shown, a student or user can connect to the on- 
line essay evaluation system via personal computer 1 10 or a similar device such as a 
10 network computer. A student may connect using either personal account 112 with a 
credit card or under school account 114. When a user logs into the system, he or she 
= --is connected to system server 116. When a user attempts to use a credit card, the 
■ credit card information is updated and verified using exterior credit card authorization 
..; ^system 118 accessed via system server 116 and database server 128. It should be 
15 ^ noted that when an account is created, it must be identified whether essays that are 
submitted under that account are to be evaluated on a regular five day schedule or a 
two day rush schedule. When the user orders the service, the user or student identifies 
the type of service which is desired, i.e. either standard 5 day turn-around or rush 2 
day turn-around. Students select an essay question, fashion a response, and enter the 
2 0 response into the system for evaluation. 

Primary readers and backup readers or evaiuators also connect to system 
server 116 via personal computer 119 or similar device. Readers rate the essays 
submitted by students and provide feedback as explained below. 

Scoring leader readers connect to system server 116 via personal 

2 5 computer 120 or equivalent device. Scoring leaders perform normal reader functions 

and may also check reader work and override a reader where appropriate. Staff of the 
scoring system service provider can connect to the system via personal computer 121 
or similar device to perform system maintenance. Various activities performed by 
administrative staff such as generating reports 122, exporting data 124, and processing 

3 0 purchase orders 126 require interaction with system database server 128. 

A preferred system architecture for an Internet based essay evaluation 
system is depicted in FIGURE 2. As shown, a user may connect to the evaluation 
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system over the Internet or through another network using web browser 140. The 
server side of the system comprises a web server 142 such as Netscape's Enterprise 
Server and Oracle database 144 running on a multitasking operating system 146 such 
as UNIX or Microsoft NT. Firewall 148 protects system resources such as legacy 

5 databases 150 and database administrative facilities 152 of the evaluation provider from 
unauthorized tampering. 

FIGURE 3 provides a diagram of the functional components of a 
preferred embodiment of the system. As shown, the system can be broken into several 
modules: administration module 160; subscription module 162; student and reader 

0 interface module 164; and reporting module 166. The administration module or essay 
response administrative system 160 comprises several subsystems: student management 
subsystem 168; reader management subsystem 170; essay management subsystem 172; 
and work distribution subsystem 174. Subscription module 162 comprises several 
subsystems: purchase order subsystem 176; credit card processing subsystem 178; and 

5 subscription-management subsystem 180. Similarly, reporting module 166 comprises 
several subsystems: e-mail notification subsystem 182; and extract subsystem 184 
which allows for sales, reader performance evaluation, and submission summaries. 
Finally, student and reader interface module 164 comprises several subsystems: 
submission subsystem or essay response submission system 186; user authentication 

0 subsystem 188; essay evaluation subsystem or automated grading system 190; and 
essay result viewing subsystem or evaluation delivery system 192. 

In the preferred embodiment, the system is implemented with a 
relational database. Of course, an object oriented database or flat file implementation 
could be used. FIGURE 4 provides a portion of the entity relationship diagram for the 

5 database data dictionary. As shown, the database of the present system comprises 
table 193 in which is stored information regarding essays submitted for scoring. Data 
related to the question and exam for which a particular question is stored in tables 194 
and 195. Information about the students who submit the essays as well as the readers 
who evaluate the essays are stored in table 196. Likewise, the rubrics and overall 

0 evaluation scores are stored in tables 197 and 198. The system provides storage for 
additional information such as reader scheduling and credit card information (not 
shown). 
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Figure 5 depicts some of the functional operations provided by the 
system. As shown, the system provides means 200 for a user or organization such as a 
school to set up accounts for submitting essays. Times and dates for which readers are 
scheduled to act as primary readers, backup readers, and reader leaders can also be 
5 entered into and retrieved from the system by means 202. Of course, the system 
provides means 204 to submit essays for evaluation. Readers use means 206 to 
evaluate essays. Means 208 to compensate readers for their efforts is also provided. 
As shown, these operations involve updating and accessing information which is stored 
on system database server 128. 

10 < Figure 6 provides an overview of the essay submission and evaluation 

process. At step 210, the student/user purchases an access code. Access codes are 
purchased for either rush or regular grading. Thus, the access code identifies an essay 
as one to be scored either on a regular 5 day turnaround or a rush 2 day turnaround. 
i-At step 212, the student submits an essay. A primary reader who has signed up for the 

15 particular reading day, logs into the system and is presented with a customized list of 
topics and questions which the reader is qualified to score. An additional icon appears 
if the reader is also qualified to serve as a scoring leader for the particular question. 
At step 214, a reader retrieves an essay from the question pool. The reader has two 
hours in which to read and score the essay. At step 216, if the reader does not submit 

2 0 the essay within two hours, the essay is returned to the pool at step 218 for another 

reader to evaluate. If at step 216 the reader completes evaluating the essay, the reader 
can continue scoring essays or can exit the system. After an essay Is evaluated, at step 
220 a reader leader has a set time frame in which he or she can override the primary 
or backup reader's assigned score. The time frame for overriding a score is 
25 configurable and is preferably set to 15 minutes. The review function may be used to 
provide critique of a reader. After an essay has been evaluated and the time frame for 
the reader leader override has expired, a student may retrieve the evaluation and 
feedback information at step 222. In the preferred embodiment, the student or user 
may retrieve his or her evaluation only if their account information indicates the bill 

3 0 has been paid. 

Figures 7 through 17 provide a view of the various screens that a user 
may encounter while using the system to submit essays. Figure 7 illustrates an 
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introductory screen through which a user may access the system functionality. As 
shown, general information regarding the system is provided. The upper left hand side 
of the screen provides linked lines of text, 230 through 239, which a user may "click 
on" with a mouse button to access various components of the user interface. 
5 A user can access directions for using the user interface of the system by 

selecting text line 230. When a user selects line 230, the user is presented with a 
screen similar to that shown in Figure 8. A user can access information regarding the 
fees for using of the system by selecting text line 231. When a user selects line 231, 
the user is presented with a screen similar to that shown in Figure 9. A user can 

10 sample the type of feedback information that is presented by the system when an essay 
is submitted by selecting text line 232. When a user selects line 232, the user is 
presented with a screen similar to that shown in Figure 10. A user can preview the 
topics for which he or she can write and submit an essay by selecting text line 233. 
When a user selects line 233, the user is presented with a screen similar to that shown 

15 in Figure 11. A user can order an access code under which an essay may be submitted 
by selecting text line 234. When a user selects line 234, the user is presented with a 
screen similar to that shown in Figure 12. A user can access information regarding 
writing an essay for evaluation by selecting text line 235 . When a user selects line 

235, the user is presented with a screen similar to that shown in Figure 13. A user 
20 can submit an essay for evaluation by selecting text line 236. When a user selects line 

236, the user is presented with a screen similar to that shown in Figure 14. A user 
can access feedback regarding an essay that he or she previously submitted by selecting 
text line 237. When a user selects line 237, the user is presented with a screen similar 
to that shown in Figure 15. A user can access responses to frequently asked questions 

2 5 about the system by selecting text line 238. When a user selects line 238, the user is 

presented with a screen similar to that shown in Figure 16. A user can access help on 
the system by selecting text line 239. When a user selects line 239, the user is 
presented with a screen similar to that shown in Figure 17. 

In the preferred embodiment, the on-line essay evaluation system may be 

3 0 accessed over the Internet. Using the above described screens, all of which may be 

accessed over the Internet with world wide web browser, a user or student is able to 
order an access code under which he or she may submit an essay or constructed 
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response. Using an access code, a user may access system functionality for submitting 
essays or constructed responses. After the essay has been evaluated, a user may enter 
the system to retrieve an evaluation of the essay that he or she submitted. All of this 
system functionality can be accessed over the world-wide web. Thus, the preferred 
5 embodiment is an Internet based on-line essay evaluation system which provides 
functionality to read an essay question, construct an essay response to the question, 
submit the essay for evaluation, and retrieve feedback information from experienced, 
trained readers. ' ' 

As noted above, the essay evaluation system provides for reader 

10 scheduling and prioritization of essay responses so that all essay responses are 

evaluated within the requested time period. In the preferred embodiment, essays are 
positioned in the queue by priority of date and time due. Thus, a rush essay which is 
due before a standard 5 day order essay, is positioned in the queue of essays to be 
read. The system automatically monitors the number of essays waiting to be read so 

15 that all essays are evaluated within the scheduled 48 hour or 5 day turnaround. When 
the system detects that essays are not being processed fast enough to meet the 
turnaround commitment, e-mails are automatically sent to the backup readers and to 
administrative staff. Alternatively, or in addition, a manual process is initiated to call 
back up readers and to schedule additional readers in order to process the backlog. 

2 0 A flow chart illustrating the essay monitoring and reader scheduling 

process is depicted in FIGURE 18. At step 250, the number of essays that need to be 
scored in the upcoming 24 hours in order to satisfy all scoring deadlines is calculated. 
At step 252, the system searches the database to identify the readers and the total 
number of each of the various question responses the readers are scheduled to read 

2 5 during the same 24 hours. If at step 254 the number of essays that need to be scored 

in the up-coming 24 hours is less than the number of essays that are scheduled to be 
read by scheduled primary readers in the same period, the system returns to step 250 
of determining the number of essays to be scored. If at step 254, however, the number 
of essays that need to be scored in the coming 24 hours is greater than the number of 

3 0 essays that are scheduled to be read by scheduled readers, the system proceeds to step 

256 where backup readers and appropriate administrative personnel are notified of the 
possible shortcoming. In the preferred embodiment, the readers and operators are 
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notified by e-mail. It is then incumbent upon the backup readers and the administrative 
personnel to insure that the appropriate essays are read so as to meet all time 
restraints. 

To insure smooth operation, the system also automatically performs 
5 workload leveling. Workload leveling guards against the situation where the first 

reader who logs onto the system "ties up" all the essays on a particular topic. Rather, 
the present system provides essay responses one at a time to readers when an essay is 
requested by the reader. When the queue of unread essays on a given topic has been 
exhausted, a message will be displayed indicating that there are no more essays on that 

10 topic. The reader then has the option of logging off or, if he or she is qualified to do 
so, switching to another topic. 

FIGURE 19 depicts the workload leveling process of the present 
invention. As shown, at step 270, the reader selects a topic area in which to evaluate 
essays. If at step 272 it is determined that essays for the particular topic and question 

15 are available to be scored, the next essay in the queue of essays to be scored is 

assigned to the particular reader at step 282. If, however, at step 272, it is determined 
that no essays on the particular topic need to be scored, the system at step 274 notifies 
the reader of such and determines which other topics the reader is authorized to score. 
If at step 276 it is determined that other topics which the reader is authorized to score 

2 0 have essays waiting to be read, the reader is allowed to select from the additional 

topics and essays. However, if at step 276 it is determined that there are no other 
essays to be read for which the reader is qualified, the system at step 280 notifies the 
reader of such. 

The system also provides for a novel method of compensating readers 
25 whereby readers in the present system are compensated for each essay that is read. 

This in contrast to the traditional reader compensation model whereby readers are. paid 
by the scheduled reading event without regard to the actual number of essays read. A 
scheduled reading event would be defined as, for example, "the AP reading" which 
would last for several days each June; in prior systems a participating reader would be 

3 0 compensated for the reading regardless of the number of essays he or she actually 

read. The compensation method of the present system correlates compensation to work 
actually performed. Because credit card rejects are so low and to meet time 
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commitments to end users, readers are paid regardless of whether or not the purchase 
is approved by the credit approval system. The system also provides a means to vary 
the amount paid for evaluating responses for different examination types or even 
% different questions in the same examination. Quality is ensured by scoring leaders who 
5 * monitor evaluations and through periodic reviews by statistical analysis of reader 
£ performance data. 

FIGURE 20 depicts a view of a process for calculating reader 
compensation. The first step in the process, step 310, involves identifying the 
. particular reader. At step 312 the total number of essays evaluated by the particular 
10 reader is accessed, usually in a database where such information is stored. At step 314 
the compensation for the particular reader is calculated as a function of the number of 
essays identified in step 312, In a simple arrangement, the compensation is equal to 
^ the number of essays evaluated multiplied by a constant rate. In more complicated 
% embodiments, the rate of compensation may vary by question. 
15 % Another novel aspect of the present system is the scoring and feedback 

functionality. In the present system, a reader selects an overall evaluation comment 
and an overall score for each essay response. Thereafter, the reader may select 
various rubrics or feedback bullets which have been pre-stored in the system for the 
particular question and thereby provide more specific feedback comments regarding the 
2 0 essay. A reader may only choose from the bullets which are provided by the system. 
In this way the system insures consistent, standardized feedback is provided to users. 
Readers may choose as many feedback bullets as applicable to the particular essay. 
The overall evaluation and the particular rubrics that may be selected for a particular 
evaluation are controlled and may be changed using system functionality. In addition 

2 5 to an overall evaluation and the rubrics, a user receives a copy of the scoring standards 

and guidelines for the particular question as well as several exemplar essay responses 
that have received various scores. In the preferred embodiment, the user does not 
receive the overall score. The overall score is kept in the system for monitoring and 
reader evaluation . 

3 0 FIGURE 21 provides a flow chart of the feedback process of the 

invention. At step 320, the reader is provided with a list of essays to score. The 
reader selects and evaluates an essay. The reader can reference various information 
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regarding scoring the particular essay such as guidelines for scoring the essay and 
sample responses with the scores that were assigned to each. At step 322, the reader is 
provided with a list of overall scores. The reader selects an overall evaluation 
comment and an overall score for the essay. At step 324, the reader may select from a 
5 predefined group of feedback rubrics or bullets. The rubrics associated with a high 
overall score tend to be positive and point out the positive aspects of the user's essay 
while the rubrics associated with a low overall score tend to point out the weaknesses 
in the user's essay and suggest methods of improving. At step 326, the information 
that will be made available to the user is linked together for convenience and access. 

10 The feedback information comprises the following items: the original question; the 

user's response; the scoring standards or guidelines for the particular question; sample 
essays that have received various scores; an overall evaluation; and a set of rubrics or 
feedback bullets. The score assigned by the reader is not provided to the user but is 
stored for use in research and monitoring readers. The final step 328, involves 

15 providing the feedback information to the user. In the preferred embodiment the 
information is accessed by the user when the user logs onto the testing web site. 
Alternatively, the information could be transmitted to the user via e-mail or some other 
means. 

FIGURE 22 provides an illustrative screen that a reader may encounter 
2 0 when logging into the system for purposes of scoring essays. As shown, the reader is 
presented with a group of topics from which they may select responses for grading. It 
should be noted that the reader will be able to access only those topics and related 
responses for which the reader has been approved. For each question in each topic 
area, the reader may select to view the training pack, grade the essay, or perform 

2 5 scoring leader functions if qualified. 

If the reader selects to view the training pack, the reader will be 
presented with a screen such as that shown in FIGURE 23. At the top of the screen 
the reader has available several items which may assist in his or her grading of the 
particular question. The reader may select to review the question that was actually 

3 0 posed to the user by selecting text item 400. The reader might also review the scoring 

guide by selecting text item 402. Alternatively, the reader may select to review sample 
essay responses that have been assigned particular scores by clicking on text item 404. 
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If the reader selects to view the question that was posed to the user, the 
reader may be presented with a screen similar to that shown at FIGURE 24. If the 
reader selects to view the scoring guideline, the reader may be presented with a screen 
similar to that shown in FIGURE 25. This screen presents information that the reader 
may find helpful in grading essays. If the reader selects to view a sample essay which 
has a particular score associated with it, the reader may be presented with a screen 
similar to that shown in FIGURE 26. By reviewing essays that may have been 
assigned various scores, the reader is able to calibrate him or herself for scoring 
recently submitted essays. 

At the screen shown at FIGURE 22, the reader may also select to score 
an essay. If the reader selects to score the essay, the system may present a screen 
similar to that shown in FIGURE 27. At the top of the screen is displayed the possible 
scores that may be assigned. The lower portion of the screen displays the essay that is 
being scored. As shown at the top of the screen, once the reader has assigned a score 
and rubrics, the reader may submit the score and quit grading or submit the score and 
continue grading. The reader may also clear any scores that he or she has assigned by 
pressing the reset button. If the reader wishes, he or she may also select to see a 
description of the rubrics that correspond to the various overall scores. The screen on 
which the description of the rubrics is provided may appear similar to that shown in 
FIGURE 28. 

The reader functionality described above, like the user functions of the 
on-line essay evaluation system may be accessed over the Internet. Using the above 
described screens, all of which may be accessed over the Internet with an Internet 
browser, a primary reader, backup reader, and reader leader can access essays or 
constructed response that have been submitted by a user. The reader may proceed to 
evaluate the essay using the evaluation guidelines and other aides that are described 
above and provided on-line. A reader may then submit an evaluation of an essay. 
Thus, the preferred embodiment of the on-line essay evaluation system provides 
Internet accessible functionality for retrieving an essay response, reviewing information 
helpful in evaluating the response, assigning an overall evaluation and rubrics to the 
essay, and submitting an essay evaluation into the system for retrieval by the user or 
student. 
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Software for performing the above described processing and presenting 
the above described screens resides primarily on system server 116 and database server 
128. The online-essay evaluation system, however, employs a client server 
technology. Thus, portions of the system software execute on the personal computers 
5 from which users and readers access the system. 

Although exemplary embodiments of the invention have been described 
in detail above, those skilled in the art will readily appreciate that many additional 
modifications are possible in the exemplary embodiment without materially departing 
from the novel teachings and advantages of the invention. For example,, those ^skilled 
10 in the art will appreciate that the screens employed may vary from those described 

above. Further, the rate of compensation for readers may vary between questions and 
test types. Accordingly, all such modifications are intended to be included within the 
scope of this invention as defined in the following claims. 
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What is claimed is: 

1. A method for on-line evaluation of a constructed response to an 
examination question, comprising the steps of: 

accessing a web site via the Internet, ordering an evaluation account, and 

selecting an examination question; 

an examinee constructing an essay response to said examination 

question; 

electronically submitting said essay response to said web site for 

evaluation; , 

positioning the submitted essay response in a queue of essay responses 

that need to be evaluated by a grader, the essay being placed in the queue based upon 

at least one of the time of submission of said essay response and the date an evaluation 

of the essay response is due to the examinee; 

the grader evaluating the essay response in accordance with calibrated 

grading guidelines, selecting an overall evaluation, one or more pre-defined feedback 

comments, and a score; and 

releasing for retrieval by the examinee the overall evaluation and the 

pre-defined feedback comments regarding said essay response. 

2. An on-line essay evaluation system for submitting essay responses 
for evaluation by graders and retrieving an evaluation of the essay response from the 
grader for submission to the examinee, comprising: 

a database which stores a plurality of questions which elicit essay 
responses from an examinee; 

an essay response submission system which selects a question from said 
database and which submits an essay response to the selected question; 

an essay response administrative system which stores essay responses 
from a plurality of examinees in a queue and selectively distributes the queued essay 
responses to graders for evaluation; 

an automated grading system which enables a grader to evaluate an essay 
response, to select an overall evaluation, one or more pre-defined additional comments, 
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and a score; and 

an evaluation delivery system which releases to the examinee for the 
examinee's retrieval said overall evaluation and any pre-defined additional comments 
from the grader. 

3. The system of claim 2 wherein said essay response administrative 
system schedules graders to evaluate queued essay responses so that essay responses 
are evaluated and feedback is stored and released for retrieval by the examinee within a 
predetermined time period. 

4. The system of claim 2 wherein said essay response administrative 
system prioritizes the presentation and distribution of essay responses to graders so that 
essay responses are evaluated and feedback is stored and released for retrieval by 
examinees within a predetermined time period. 

5. The system of claim 2 wherein said essay response administrative 
system automatically calculates any compensation for the graders based upon the 
number of constructed responses the grader has evaluated. 

6. The system of claim 2 wherein said automated grading system 
enables a grader leader to view the overall evaluation and pre-defined additional 
comments of the grader and assign a new overall evaluation and pre-defined additional 
comments if desired. 

7 . The system of claim 2 wherein said evaluation delivery system 
releases to the examinee for the examinee's retrieval at least one exemplary essay 
response, said overall evaluation, and said pre-defined additional comments. 

8. The system of claim 2 wherein said essay response administrative 
system includes software which: 

identifies the number of essay responses which need to be evaluated in 
an upcoming time period; 
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identifies the number of essay responses which are scheduled to be 
evaluated in the upcoming time period; 

if the number of essay responses that need to be evaluated in the 
upcoming time period is less than the number of essay responses that are scheduled to 
be evaluated in the upcoming time period, identifies the number of essay responses 
which need to be evaluated in the upcoming time period; and 

if the number of essay responses that need to be evaluated in the 
upcoming time period is greater than the number of essays that are scheduled to be 
evaluated in the upcoming time period, electronically notifies backup graders of the 
need to provide assistance in the upcoming time period. 

^ 9. The system of claim 2 wherein said essay response administrative 

system includes software which: 

v identifies the date and time an essay response was submitted; 

identifies whether the essay response is to be graded on a regular 
schedule or a rush schedule; 

calculates a date for which an evaluation of the essay response is due 
based upon the date the essay response was submitted and whether the essay response 
is to be evaluated on a regular schedule or a rush schedule; and 

places the essay response in a queue of essay responses that is ordered 
by the date on which an evaluation of each response is due. 

10. The system of claim 2 wherein said essay response administrative 
system includes software which: 

identifies a grader; 

identifies the total number of essay responses read by the grader; and 
calculates the compensation for the grader as function of the number of 
essay responses read. 

11. The system of claim 2 wherein said essay response 
administrative system includes software which: 

identifies that a grader has submitted an overall evaluation and score for 
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an essay response; 

forwards the essay response and overall evaluation to a grader leader for 

review; and 

if the overall evaluation and score are not consistent with scoring and 
5 evaluation standards, allows grader leader to assign a new score and evaluation, so 
long as the supervisory grader performs the review in a predetermined time period. 



12. The system of claim 2 wherein said automated grading system 
presents an essay response to a grader so that the grader may evaluate the essay 
response; and presents evaluation categories to the grader so that the grader may select 

10 an overall evaluation. 

13. The system of claim 2 wherein said evaluation delivery system 
groups the examination question, scoring guidelines, sample essay responses, the 
overall evaluation, and pre-defined additional comments into a feedback package; 
stores said feedback package; and releases said feedback package for retrieval by the 

15 examinee. 



14. An Internet based system for submitting and evaluating essay 
responses to examination questions, comprising: 

a server computer which forwards examination questions to examinees 
and accepts essay responses to said examination questions for evaluation, said server 
2 0 being connected to the Internet; 

a first computer connected to said server computer so as to permit 
examinees to set up an account for purchasing access to an examination question and 
evaluation of an essay response submitted in response to said examination question; 
and 

2 5 a second computer connected to said server computer so as to permit 

graders to read an essay response that was submitted by an examinee, evaluate the 
essay response by providing an overall comment and pre-defined additional comments, 
store said overall comment and said pre-defined additional comments, and release said 
overall comment and said pre-defined additional comments for retrieval by the 
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examinee via said server computer. 

15. A method for on-line evaluation of constructed responses to an 
examination question over a system having a server computer connected to the Internet, 
a first computer electronically connected to said server computer, and second computer 
5 connected to said server computer, comprising the following steps: 

an eixaminee accessing the server computer from said first computer to 
set up an account for purchasing access to an examination question and evaluation of 
an essay response which is submitted in response to said examination question; 

an examinee accessing the server computer from said first computer to 
10 ^submit an essay response in response to said examination question; 

accepting said essay responses at the server computer; 
~ a grader accessing the server computer from the second computer to 
read the essay response submitted by the examinee, evaluate the essay response by 
providing an overall comment and pre-define;d additional comments, storing said 
15 overall comment and pre-defined additional comments, and releasing said overall 
comment and pre-defined additional comments for retrieval by the examinee. 
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SAT II: Writing 

The Online Essay Evaluation Service works best with Netscape 2.0 
or better and Microsoft Internet Explorer 3.0 or better. To download 
the latest browser version, select the appropriate button. 



Get ready for SAT II: Writing Test questions with the Online Essay 
Evaluation Service. You'll have a chance to try real test questions and 
to get valuable feedback to help you prepare for the actual test day . 

Here are some of the reasons the Online Essay Evaluation Service 
will help you do your best when you sit down to take an SAT II: 
Writing Test: 

□ Real Questions: The service provides you with the only online 
opportunity to prepare for the SAT II: Writing Test using actual 
questions from recent tests. jt t 

a Valuable Feedback: You get useful, instructive feedback to 
~ help ybii improve your essay as you prepare for test day. 

□ Experience: Because you are practicing on questions written by 
SAT test developers and receiving evaluations from the readers 
of the actual exams, you'll learn what to expect and how to 
prepare for test day. 

What Yoii Receive 

Within five business days, your personal essay evaluation will become 
available to you on this web site. Your evaluation will include: 

o a copy of the guidelines used to score your essay; 

□ one or more model essays written by successful test takers, along 
with reader's commentary; 

a helpful feedback on the strengths and weaknesses of your essay; 
and 

a an, overall evaluation. 

The evaluation you Will receive is not a predictor of any future 
performance on the SAT II: Writing Test. Additionally, to ensure 
the fairness of scoring, and to evaluate and improve this service, 
your practice essay, without your identifying information, may 
also be used for research purposes. 



Directions 

Informatio n on the SAT TI Writing Test 



This entire site protected bv copyright All rights reserved. By accessing and using 
this site, you agree to be subject to the "Torms and Conditions Qovfrmn? Use flfftf 
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SAT II: Writing 
Directions 



5. 

6. 
7. 



9. 



10. 
11. 



Sample: Take a look at sample topics with sample student 
essays and evaluations. 

Pick a Topic: Preview (before paying for anything) questions 

that you want to answer and have evaluated by trained readers. 

Print the test question(s) you will answer. 

Order: Receive an access code for each question you select 

Students must have a major credit card: VISA, MasterCard, or 

American Express. Schools must have a credit card or purchase 

order. 

You will receive your access code(s) on the web site 
immediately after completing the order form. 
Print and keep a copy of your access code(s). 
Write: Prepare your essay under conditions similar to the real 
test experience. 

Submit: Use your access code(s) to send your essay in for 
review. 

Get Evaluation: Return to the web site within five days after 
submission to get your evaluation. You will need your access 
code. 

FAQ: If you have questions, review frequently asked questions. 
Help: If you need additional assistance, select Help. 



Fees 



This entire site protected by copyright. All rights reserved. By accessing and using 
this site, you agree to be subject to the "Terms and Conditions Governim* Use and 
Access to C ollege Board Online." 
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SAT II: Writing 
Fees 

Students and schools can use the Online Essay Evaluation Service. 
To purchase, students need a VISA, MasterCard, or American Express 
card. Schools can use a purchase order or a credit card. 

Students may purchase SAT II: Writing essays in two ways: 

Z2 one practice essay at a time for $10 per essay 
g three practice essays at a time for $25 total 

Schools can order multiple practice essays as listed (or in multiples of 
these quantities): 

a ten practice essays at a time for $85 total 

□ thirty practice essays at a time for $255 total 
c fifty practice essays at a time for $400 total 

□ one hundred practice essays at a time for $800 total 

Once you complete the order form vou will immediately receive an 
access code on this web site for each evaluation you ordered You may 
use an access code to submit a response to any of the available SAT II: 
Writing questions. 

Access codes are good for one-time use, for one year. 

After writing and submitting your essay, your personal evaluation will 
be available on this web site within five working days. 

Rush order: You may also choose to have your evaluation(s) rushed 
back to you within two business days for an extra $12.50 per essay. 

To Order 

Sample SAT IT; Writing question, essays, anfl valuations 



This entire site protected by copyright. All rights reserved. By accessing ^fing 
this site, you agree to be subject to the "Tnrm* mid Conditions Gowrnm? Use and 
Annex* to Co\l<>™ Board Online. " 
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SAT II: Writing 

Samples of What You Receive 

Below are samples of what they receive once their SAT II: Writing 
essay has been evaluated: 



a The question to which they responded 
c Their sample essay 

d Their essav evaluation 

c The Scoring Guide used by readers 

- Sample model essays, " excellent " and "good" 

Pick a Topic 



This entire site protected bv copyright. All rights reserved. By accessing and using 
this site, you agree to be subject to the "Terms and Conditions Governin? Use ana 
Access to C nlleve Board Online. " 
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SAT II: Writing 
Pick a Topic 

Preview the topics listed below before choosing the essay(s) you 
would like to write. When you have chosen a topic, pnnt out the 
relevant pages and then do your writing. This service lets you write on 
any essay topic before you actually pay for the evaluation. 

Available Topics 

G Reputations 

□ New Ideas 

g ^'Significant Event 

3 Hematic Results 

g On the Shoulders of Giants 



Order 



This entire site protected by copyright Alt rights reserved. By accessing and using 
this site, vou agree to be subject to the "Tnrm* and C onditmm Oovernm? Use and 
Aneexx m C nilvve Board Onlim. " 
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SAT II: Writing 
Order 

Place your order for the Online Essay Evaluation Service. You will 
receive one access code for each evaluation you order. Access codes 
are required to submit essays. 

Students must have a major credit card: VISA, MasterCard, or 
American Express. 

Schools must have a credit card or purchase order. 
For information on the fees for this service, see: 
SAT 11 r Writing Fees 



This entire site protected by copyright. All rights reserved. By accessing and using 
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SAT II: Writing 

How to Write Your Essay 

Option #1 By Hand - In order to best prepare for the SAT II: Writing 
Test, we recommend writing your essay under normal exam-like 
conditions. Use a pen and paper and handwrite your essay, observing 
the time limits for planning and writing. Then, you can type your essay 
on a computer using a standard word processing program, save it^ and 
copy and paste it into the Submit page of this web site. 

Option #2 By Computer or Word Processor If you prefer, you can 
type your essay on a computer using a standard word processing 
program, save it, and copy and paste it into the Submit page. ■ 
Remember, you won't be able to type your essay during the actual 
exam! < . . . . . 

Option #3 Directly Online - You can also type your essay directly 
onto the Submit Essay web page, but we do not recommend it. If your 
web access is interrupted, you may lose your essay and have to recreate 
your answer. A very frustrating process! 

When writing your evaluation essay, follow these tips to become better 
prepared for the SAT II: Writing Test: 

1 Whether vou decide to handwrite your essay first or just type it, 
be sure to* save your essay on the computer, and save it often, in 
the event of any problems. 

2. For the most authentic evaluation, do not use the grammar ana 
spelling check option on your word processing program. You 
won't be able to use one during the test. 

3 . Remember that in order to receive the most helpful feedback you 
should compose your essay under exam-like conditions, 
observing the specified time constraints for planning and writing. 

Use your browser's back button to return to a topic or select from the 
menu at left. 



This entire site protected by copyright. All rights reserved. By accessing and using 
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Online Essay « gf§L 
Evaluation Service 

sat h; writing Submitting Your Essay 

£^ Ct1 ° nS Please select the essay you are submitting. Then complete the 

Samples information below. 

Pirk a Tonic 

AP Biology 

sSbn^t Properties of Water 

riPt Rvatuation Animal Systems Structure and Function 

FAQs Laboratory Question: Plant Growth 

HfiUa Mitosis/Meiosis 

p«..™-tft ^tart Recycling of Oxygen 

Rn, i nm ^ m Regulatory Mech^isms 

c.nWe pe Board Online Laboratory Question: Animal Behavior 

Site Search Laboratory Question: Genetics and Hardy- Weinberg 

AP English Language and Composition 
Lady Montagu 

Gary Soto's Autobiographical Narrative 

Lewis Lapham's Money and Class in America 

Meena Alexander's Fault Lines 

Frederick Douglass 

1984 and Brave New World 

AP English Literature and Composition 

The House of the Seven Gables 

Anne Bradstreet Poem 

Open Essay Question on Happy Endings 

On The "Death of a Toad" Poem 

Obasan 

Open Essay on Social Occasions 

AP U. S. History 

American Revolution 
Articles of Confederation 
Jacksonian Democracy 
Territorial Expansion 

Reconstruction Document Based Question 

Consequences of the Civil War 

Women, 1890-1925 Document Based Question 

McCarthyism 

SAT II Writing 

Reputations 

New Ideas 

A Significant Event 

Dramatic Result FIGURE 14 

On the Shoulders of Giants 

Identification Information 
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Get Your Feedback 

Please enter your Access Code: 
Access Code: 



This entire site protected by copyright. All rights reserved By accessing and 
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SAT QUESTIONS 

Q. When should I plan to write a SAT II: Writing Test practice 
essay? 

A. Depending on when you plan to take the actual test, it would be 
most useful to try the Online Essay Evaluation Service when you can 
use the feedback you receive to improve your writing and performance 
on the test. If you plan to test in the spring, you may want to try the 
service early in the school year. This gives you plenty of time to review 
your evaluation and to work on your weaknesses before you take the 
actual tests. 

Q. Is there a limit to how many times I can write and submit the 
same essay for evaluation? 

A* No. 

Q. Does the College Board recommend writing the same SAT II: 
Writing Test practice essay more than once? 

A. It might be helpful to write the same essay twice to find out if you 
improved your previously evaluated weak areas. However, since you 
won't have the opportunitv to write the same essay on the actual test or 
examination, it may be more helpful to take two comparative essays. 

Q. Is there a deadline for submitting an essay once I've ordered 
and received the access code? 

A. Yes, you must submit the essay within one year after purchasing the 
practice essay. 

Q. When will my evaluation be ready after it's submitted? 

A. Your evaluation should be available within five business days. 
However, because the evaluations are released as soon as they are 
completed, you may want to check the site after three or four days. 

Q. Who should take the SAT II: Writing Test? 

A. Some colleges require SAT II: Subject Tests for admission. 
Colleges use the tests to help determine how well prepared you are tor 
different college programs, place vou in freshman and higher-level 
course work, and advise vou on course selections. Subject Test scores 
can help you demonstrate your achievements. Check directly with the 
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SAT II: Writing 

Try FAQs for answers to questions on the S AT II: Writing part of the 
service. If these responses do not help, email us at 
ONTmF-5ftSAVffiets.org so we can help you. 



This entire site protected by copyright. All rights reserved By accessing and using 
this site, you agree to be subject to the "Terms and Conditions Gmvrrmnv Use and 
Arms in Cg/ir?* > Rnard Online. " 
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AP 1996 English Language 
Question 1 Training Pack 

^400 

This training pack contains a copy of the question you are reading, the 
402 — s m -yn g fuide used at the reading, and sample papers at score points L2« 2* 
4»5.&2*∧L 404 



AP English Language Question 1 

In the following passage from a letter to her daughter, Lady Mary Wortley 
Montagu (1689-1762) discusses the education of her granddaughter. 

Read the passage carefully. Then write an essay in which you analyze how 
Lady Mary uses rhetorical strategies, and stylistic, devices to convey her 
views about the role knowledge played in the lives of women of her time. 

True knowledge? consists in knowing things, not words. I 
would Wish her no further a linguist than to enable her to read 
books in their originals, that are often corrupted, and always 
injured, by translations: Two hours' application every morning 
will bring this about much sooner than you can imagine, and she 
will have leisure enough besides to run over the English poetry, 
which is a more important part .of a woman's education than it is 
generally supposed. Many a young damsel has been ruined by a 
fine copy of verses, which she would have laughed at if she had 
known it had been stolen from Mr. Waller. I remember, when I 
was a girl, I saved one of my companions from destruction, who 
communicated to me an epistle she was quite charmed with. As 
sh6 had a natural good taste, she observed the lines werenot so 
smooth as Prior's or Pope's, but had ; more thought arid spirit than 
any of theirs. She was wonderfully delighted with such a 
demonstration of her lover's sense and passion, and not a little 
pleased with her own charms, that had force enough to inspire 
such elegancies. In the midst of this triumph I showed her that 
they were taken from Randolph's poems, and the unfortunate 
transcriber was dismissed with the scorn he deserved. To say 
truth, the poof plagiary was very unlucky to fall into my hands; 
that author being no longer in fashion, would have escaped any 
one of less universal reading than, myself.. You should encourage 
your daughter- to talk over with you what she reads; and, as you 
are very capable of distinguishing, take care she does not 
mistake pert folly for wit and humour, or rhyme for poetry, 
which are the cdnimon errors of young people, and have a drain 
of ill consequences. The second caution to be given her (and 
which is most absolutely necessary) is to conceal whatever 
learning she attains with solicitude . . . ; the parade of it can only 
serve to draw on her the envy, and consequently the most 
inveterate/hatred, of all he and she fools, which will certainly be 
at least three parts in four of all her acquaintance. The use of 
knowledge in our sex, besides the amusement of solitude, is to 
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AP English Language Question 1 

In the following passage from a letter to her daughter, Lady Mary Wortley 
Montagu (1689-1762) discusses the education of her granddaughter. 

Read the nassage carefully. Then write an essay in which you analyze how 
Lady Mary uses rhetorical strategies and stylistic devices to convey her 
views about the role knowledge played in the lives of women of her time. 

True knowledge consists in knowing things,not words. I 
would wish her no further a linguist than to enable her to read 
books in their originals, that are often corrupted, and always 
injured, by translations. Two hours' application every morning 
will bring this about much sooner than you can imagine, and she 
will have leisure enough besides to run over the English poetry , 
which is a more important part of a woman's education man it is 
generally supposed. Many a young damsel has been ruined by a 
fine copy of verses, which she would have laughed at if she had 
known it had been stolen from Mr. Waller^I remember when I 
was a girl, I saved one of my companions from^destruction who 
communicated to me an epistle she was quite charmed with. As 
she had a natural good taste, she observed the lines were not so 
smooth as Prior's or Pope's, but had more thought and spirit than 
any of theirs. She was wonderfully delighted with such a 
demonstration of her lover's sense and passion, and_not a little 
pleased with her own charms, that had force enough to inspire 
such elegancies. In the midst of this triumph I showed her that 
they were taken from Randolph's poems* and the unfortunate 
transcriber was dismissed with the scorn he deserved. To say 
truth, the poor plagiary was very unlucky to fall into my hands; 
thatiuthor being no longer in fashion, would have escaped any 
one of less universal reading than myself. You should encourage 
your daughter to talk over with you what she reads; and, as you 
are very capable of distinguishing, take care she does not 
mistake pert folly for wit and humour, or rhyme for poetry, 
which are the common errors of young people, and have a tram 
of ill consequences. The second caution to be given her (and 
which is most absolutely necessary) is to conceal i whatever 
learning she attains with solicitude . . . ; the parade of it can only 
serve to draw on her the envy, and consequently the most 
inveterate hatred, of all he and she fools, which will certainly be 
at least three parts in four of all her acquaintance. The use ot 
knowledge in our sex, besides the amusement of solitude, is to 



FIGURE 24 

25/29 



SUBSTITUTE SHEET (RULE 26) 



WO 98/43223 




PCT/US98/05804 



moderate the passions, and learn to be contented with a small 
expense, which are the certain effects of a studious life; and it 
may be preferable even to that fame which men have engrossed 
to themselves, and will not suffer us to share. 



Scoring Guide 
AP English Language Question 1 

General Directions: This scoring guide will be useful for most of the essays 
you read. When it seems inappropriate for a particular essay, consult your 
Table Leader. Also consult your Table Leader about books that seem to have 
no response or responses unrelated to the question (scored-). 

The score you assign each essay should reflect your judgment of its quality 
as a whole. You should reward the writers for what they do well in response 
to the question. Remember that students had 40 minutes to read and write. 
The resulting essays should thus be thought of as comparable to essays 
produced in final exams, not judged by standards appropriate for out-of-class 
writing assignments. 

All essays, even those scored 8 and 9, are likely to exhibit occasional flaws 
in analysis or in prose style and mechanics; such lapses should enter into 
your holistic judgment of the essay's quality. In no case, however, may an 
essay with many distracting errors in grammar and mechanics be scored 
higher than 2. 

9: Essays earning a score of 9 meet all the criteria for 8 papers and, in 
addition, are particularly full or apt in their analysis or demonstrate 
particularly impressive stylistic control. 

8- Essays earning a score of 8 effectively analyze how rhetorical strategies 
and stylistic- devices help Lady Montagu convey her views about the role 
knowledge played in women's lives. They refer to the text, directly or 
indirectly, anci are likely to cogently describe how strategies such as tone, 
irony, ^nd use of examples contribute to the passage. Their prose , 
demonstrates an ability to control a wide range of the elements of effective 
writing but is not riecessOTly fliawless. 

7: Essays earning a score of 7 fit the descriptions of 6 essays but employ 
more comfjiete analysis or more mature prose style. 

6* " Essays earning a score of 6 adequately analyze how rhetorical strategies 
and stylistic devices help Lady Montagu convey her views about the role of 
knowledge in women's lives. They refer to the text; directly or indirectly, 
and they may discuss or implicitly recognize features and devices such as 
irony, tone, and use of example as ways of demonstrating these views. A few 
lapses in diction or syntax may be present, but generally the prose of 6 
essays conveys their writers 1 ideas clearly. 

5: Essays earning a score of 5 analyze rhetorical strategies used by Lady 
Montagu to convey her views about the role of knowledge in women's lives, 
but their development of those strategies is limited. They may treat strategies 
in superficial ways or develop their ideas inconsistently. A few lapses in 
diction or syntax may be present, but usually the prose in 5 essays conveys 
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Sample Essays 

Essay TTT, Score=9 

In all great works of fiction, it is one of the author's major 
responsibilities to establish definite characteristics and shades to 
each character; how this is done is solely based on the author's 
creative perception. In his novel The House of the Seven Gables. 
Nathaniel Hawthorne utilizes an indirect description 01 the 
character of Judge Pyncheon to reveal his characteristics and 
personalities. Mainly through the usage of tone and point of 
view, Hawthorne reveals Judge Pyncheon as a proud, 
hypocritical, self-righteous, and generally reprehensible 
individual. . , , 

Overpowering the entire passage is Hawthorne s heavy, 
sardonic tone. From the outset, the narrator lavishes praise on 
Pyncheon's "purity of judicial character," "cleanliness of moral 
deportment," and "faithfulness of public service." Yet this is 
done at such an extent that this praise is rendered overblown and 
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AP English Literature and Composition - The House of the Seven Gabies 

Description 



Warning: In order to transfer any selected feedback you must use the OK button to return to the student 
essay. 



9-8 

D 1 . You provide a clear analysis of Judge Pyncheon's character, with apt and specific references to 
the text. 

□ 2. Your essay is well-developed. 

□ 3. Your essay is well-organized. 

□ 4. Your ideas are perceptive arid demonstrate an understanding of the text. 

O 5 . The essay acknowledges and addresses the complexity of Pyncheon's character. 

□ 6. Your essay discusses several literary devices that help to create Pyncheon's character. 

□ 7. You demonstrate consistent control over the elements of effective composition. 

□ 8. The essay contains only minor grammatical, mechanical, or usage errors that do not detract 

from the overall impression. Nonetheless, you should save a little time to reread your AP essays 
after you write them. 



7-6 

□ 9. You provide a generally clear analysis of Judge Pyncheon's character with some apt support 

from the text. More thorough support would strengthen the essay. 

□ 1 0. Your essay is accurate in discussing the literary techniques that bring Judge Pyncheon's 

complex character to life. Reread the passage and look for additional clues to Pyncheon's 
complexity. 

□ 11. Your essay is clear. If you are able to make it more incisive and precise, it will score even 

higher. 

□ 12. You demonstrate generally consistent control over the elements of effective composition. 

Practice in writing should help you gain even more control. 

□ 13. The essay contains minor grammatical, mechanical, or usage errors that only occasionally 

detract from the overall impression. Nonetheless, eliminating these errors will improve your 
score. Save a little time to reread your AP essays after you write them. 



5 

□ 14. While you analyze the character of Judge Pyncheon, your essay would be improved with more 

specific references and more fluid development. 

□ 15. You recognize that the narrator does not approve of Pyncheon, but the essay does not capture 

the complexity of the character. Consider the narrator's use of irony and how the narrator 
conveys certain contradictions. 

□ 16. You offer an analysis of how literary devices are used in the characterization. The essay would 

be stronger if the analysis were more penetrating. 
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